The scale_factor ensures that the initial latent space on which the diffusion model is operating has approximately unit variance. Hope this helps :)

Perfect @rromb, yes, I was looking for the principle which led to one number versus another. (Sec. 4.3.2 and Appendices D.1 and G, for anyone looking.)

To make sure I'm understanding, it sounds like you arrived at scale_factor = 0.18215 by averaging over a bunch of examples generated by the vae, in order to ensure they have unit variance with the variance taken over all dimensions simultaneously? And scale_factor = 1 / std(z), schematically?

And if the above is right, I'm curious if you also tried instead whitening each latent individually, rather than using a single global scale for all latents? Or tried using LayerNorm or similar?